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Abstract 


This paper presents new estimates of the effects of the City University of New York (CUNY) 
Accelerated Study in Associate Programs (ASAP) model, evaluated using a randomized con- 
trolled trial first in New York and later through a replication in Ohio. It describes longer-term 
effects of CUNY ASAP in New York, showing that the program’s effects on associate’s degree 
receipt persisted through eight years and likely represent a permanent increase in degree receipt. 
The paper also offers an analysis from the pooled study samples in New York and Ohio. The 
findings indicate that the program has consistent effects on degree receipt across different states 
but also for somewhat different levels of service contrast, such as the number of additional advis- 
ing visits. 
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Introduction 


Postsecondary education provides an important stepping-stone to well-paying jobs and the mid- 
dle class for many low-income individuals. The nation’s community colleges play a vital part 
in that effort, serving about 40 percent of all postsecondary students and a disproportionate 
number of low-income students. Community colleges are accessible, typically with open ad- 
mission policies, and they are, on average, much less expensive than four-year colleges or pri- 
vate two-year colleges. 


Unfortunately, however, graduation rates from community colleges are low. Among 
first-time, full-time, degree-seeking students entering public two-year schools, only 25 percent 
graduate within three years.' Policymakers and higher education administrators across the nation 
are striving to improve those rates. Such increases would benefit students and society in an econ- 
omy that increasingly demands a highly educated workforce. 


Numerous reforms have been designed to increase students’ persistence in and comple- 
tion of community college (see, for example, Angrist, Lang, and Oreopoulos, 2009; Attewell, 
Heil, and Reisel, 2012; Bettinger and Baker, 2014; Bettinger, Long, Oreopoulos, and San- 
bonmatsu, 2012; Goldrick-Rab, Harris, Kelchen, and Benson, 2012; Patel, Richburg-Hayes, de 
la Campa, and Rudd, 2013; Richburg-Hayes et al., 2009; Weiss, 2019). These policies often in- 
clude a few changes to business as usual and typically last for one or two semesters. Evaluations 
have indicated that many of these reforms led to modest improvements in student progress and, 
occasionally, had limited effects on graduation rates. 


Among the tested policies, one program model stands out: the City University of New 
York (CUNY) Accelerated Study in Associate Programs (ASAP). Designed and implemented by 
CUNY to help more students graduate, and more quickly, ASAP is more intensive and lasts 
longer than most community college reforms. It was evaluated at three CUNY colleges using a 
randomized controlled trial that started in 2009. The evaluation found that the program led to a 
substantial boost in the number of students who graduated—increasing three-year graduation 
rates by 18 percentage points. This exceptional result prompted three Ohio community colleges 
(at four campuses) to replicate the program. 


Three-year findings from the test in Ohio also showed a large increase, of 16 percentage 
points, in graduation rates (Miller, Headlam, Manno, and Cullinan, 2019). This outcome was 
encouraging, given that the replication of findings from a given test of an educational intervention 
is the exception rather than the norm (see, for example, Camerer et al., 2018). The findings from 
Ohio showed not only that the model could be implemented with a high degree of fidelity in 
another setting but that it could have similarly positive and large effects. 


This paper extends the earlier results in several ways. First, it provides an update of the 
findings from CUNY ASAP through eight years. An earlier paper documented effects through 
Year 6 and found that the impact on degree receipt was still large, at 10 percentage points. 


'See McFarland et al. (2019). Graduation rates include students receiving associate’s degrees or certificates 
from their initial institution of attendance only. 


However, the impact on degree receipt had diminished relative to Year 3, as the control group 
“caught up” somewhat to the program group (Weiss, Ratledge, Sommo, and Gupta, 2019). It was 
expected that degree receipt would increase to some extent for students in the control group, mir- 
roring national trends in which many graduates take longer than three years to earn a degree. By 
tracking effects for an additional two years, this study examines whether the effects persist or 
diminish further. The longer follow-up period also provides a more comprehensive look at the 
program’s effects on enrollment in four-year colleges and the receipt of bachelor’s degrees. 


Second, this paper synthesizes the three-year findings from CUNY and the Ohio replica- 
tion. The pooled analysis supports the estimate of a broader treatment effect (the mean effect 
across six colleges from two states) of the ASAP model on three-year graduation rates. Also, the 
pooled sample is better positioned than the individual studies to examine subgroup effects, or 
effects for different types of students. The synthesis also provides the opportunity to look at ef- 
fects across the two states and across the individual colleges. It allows for an examination of 
whether there is variation in the effects on degree receipt in different locations, and, if so, whether 
this variation is associated with differences in students’ use of program services. 


The findings from both analyses help inform a national scaling of the ASAP model. For 
example, the Ohio colleges implemented the model somewhat differently than CUNY did, and 
the impacts on service contrast varied across states and across individual colleges. The findings 
therefore speak to whether the large effects on degree receipt are robust to modest variations in 
the planned model, its actual implementation, and the counterfactual conditions against which it 
is being compared. The longer-term findings also help to assess whether the ultimate effect of 
ASAP is to speed up degree receipt and whether it leads to a permanent increase in graduation 
rates, helping students graduate who would not have done so without the program. 


In sum, the findings show that ASAP’s effect on the receipt of associate’s degrees per- 
sisted through eight years, although it did not lead to a long-term increase in the receipt of bach- 
elor’s degrees. The effects on the receipt of associate’s degrees indicate that the program not only 
helped students graduate faster but also helped students graduate who would never have done so 
without the additional support. The effects across both states through three years provide further 
evidence of the program’s effectiveness, for students in general and for different types of students. 
Finally, the effects by state and by campus suggest that there is a positive association between 
service contrast, or effects on advising visits, for example, and degree impacts. However, beyond 
some level, there may be diminishing returns. In other words, increasing advising by 15 visits per 
semester may not lead to much larger effects on degree receipt than increasing advising by 8 visits 
per semester. Because the model does cost more than the usual services that are offered to stu- 
dents, colleges will need to compare the resources needed to generate increases in service contrast 
with the additional degrees the expanded services generate. More broadly, the consistency of 
findings across the states and colleges suggests that the results are likely to generalize beyond 
these students and colleges, furthering the idea of ASAP as a national model to increase educa- 
tional attainment. 


Background 


Researchers have identified a number of student-level challenges and institutional practices that 
underlie the low rates of persistence and completion at community colleges (for overviews, see 
Baum, Kurose, and McPherson, 2013; Braxton, 2002; Calcagno, Bailey, Jenkins, Kienzl, and 
Leinbach, 2008). These factors include financial challenges, such as the costs of attending col- 
lege; a shortage and underuse of student support services; insufficient preparation for college- 
level work; and the competing demands of work, family, and school. These challenges are even 
more pronounced for low-income, less academically prepared students, who are less likely than 
other students to stay on track and graduate (Attewell et al., 2012; Crosta, 2014; Klempin, 2014). 


Although tuition and fees at community colleges are about one-third the cost of public 
four-year colleges and universities (Baum and Ma, 2014), they still constitute a substantial invest- 
ment for many low-income students, especially when opportunity costs are considered. In addi- 
tion, financial aid sometimes does not cover the full cost of attendance, leaving students to strug- 
gle to afford tuition, transportation, or textbooks. Moreover, low-income students must negotiate 
a complex financial aid system in order to receive assistance (Bettinger et al., 2012). 


Many students are not academically prepared for college, as indicated by the fact that 
nearly 60 percent enroll in at least one developmental (or remedial) reading, writing, or mathe- 
matics course during community college (Bailey, 2009). Placement into developmental courses 
is negatively associated with student persistence and success, as developmental students fre- 
quently drop out of their classes or leave college entirely, and those who remain make progress 
relatively slowly (Adelman, 2004; Attewell, Lavin, Domina, and Levey, 2006). Only 13 percent 
of community college students with developmental needs earn a certificate or degree within three 
years (U.S. Department of Education, n.d.). 


Upon entering college, many students also struggle to navigate the unfamiliar environ- 
ment and need help figuring out which courses to take and in what order, how to register for 
classes and apply for financial aid, and what resources are available to help them make the tran- 
sition to college (Bailey, Jaggars, and Jenkins, 2015). Community colleges are rarely able to sup- 
port the kind of personalized and timely advising and counseling that students need—services 
that are often provided at selective four-year colleges (Bound and Turmer, 2007). For example, 
the National Academic Advising Association (2011) estimated that the median caseload of an 
adviser at a public two-year college is 441 students per adviser, compared with 260 students per 
adviser at a public four-year college; as such, the amount of advising students receive in commu- 
nity colleges is limited. Students who do not have access to an informed adviser may neglect to 
register for a required course, fail to secure the maximum financial aid for which they qualify, or 
make other missteps that could negatively affect their progress through college. Community col- 
lege students, most of whom commute, are also less likely to identify with their college commu- 
nity than are traditional undergraduates at four-year institutions (Tinto, 1997). 


Taken together, the numerous barriers that low-income students face contribute to their 
low college completion rates. While research has identified causal evidence about the efficacy of 
postsecondary interventions, most implemented policies have had, at best, only small effects. For 


example, several studies have produced experimental evidence that financial aid—related reforms 
can positively, although often modestly, influence students’ academic progress (for example, see 
Angrist, Autor, Hudson, and Pallais, 2014; Angrist et al., 2016; Angrist et al., 2009; Angrist, 
Oreopoulos, and Williams, 2010; Bettinger et al., 2012; Cohodes and Goodman, 2014; Deming 
and Dynarski, 2010; Goldrick-Rab, Kelchen, Harris, and Benson, 2016; Mayer, Patel, and 
Gutierrez, 2015). 


Colleges have also tried many strategies to better integrate new students, such as creating 
learning communities and implementing student success courses to foster connections within the 
classroom (Engstrom and Tinto, 2008; Rutschow, Cullinan, and Welbeck, 2012; Weiss, Mayer, 
et al., 2015; Weiss, Visher, Weissman, and Wathington, 2015). Other interventions are focused 
on helping community college students stay enrolled continuously, including in summer sessions 
(Attewell and Douglas, 2014; Attewell and Jang, 2013; Weiss, 2019). Experimental evaluations 
of these approaches indicate that they can help students better acclimate to college, and some have 
produced modest improvements in academic outcomes. 


Enhanced academic advising interventions are another tactic. A fairly robust experi- 
mental literature has found that a variety of modes of advising have positive, although often lim- 
ited, effects on students’ academic outcomes (for examples, see Avery, Howell, and Page, 2014; 
Barr and Castleman, 2017; Bettinger and Baker, 2014; Carrell and Sacerdote, 2013, 2017; Evans, 
Kearney, Perry, and Sullivan, 2017; Scrivener and Weiss, 2009). 


Two more recently studied programs provide more comprehensive and longer-term sup- 
ports for students and have led to larger effects. One study is of the Early College High School 
initiative, a model that offers dual high school and college enrollment and a variety of supports to 
help students move into and complete college. Students in the program were substantially more 
likely to earn an associate’s degree than students who were not in the program (Berger, Turk- 
Bicakci, Garet, Knudson, and Hoshen, 2014). A second example comes from an evaluation of the 
Stay the Course program at a community college in Texas, which offered three years of ongoing 
comprehensive case management, provided by trained social workers, along with access to emer- 
gency financial assistance. The evaluation found large effects on degree receipt among the 20 
percent of the study sample that took up the program (treatment-on-the-treated effects), although 
the effects were primarily for women (Evans et al., 2017).* The effects among program partici- 
pants contain a fair amount of sampling error, given the small sample size. 


Thus, the weight of the evidence suggests that short-term, light-touch interventions are 
typically not enough to substantially improve longer-term outcomes, such as degree receipt, and 


For example, the offer of the program increased associate’s degree receipt among the full sample of women 
by 7.4 percentage points. The increase in degree receipt among women who participated in the program (treat- 
ment-on-the-treated effects) was 32 percentage points. 


that more comprehensive interventions that address multiple barriers are needed.* ASAP is one 
such program. 


The ASAP Model and Implementation 


The ASAP Model 


Developed by the City University of New York, ASAP is a comprehensive program that 
provides students with up to three years of financial and academic support and other support ser- 
vices to address multiple barriers to success, with the goal of helping more students graduate 
within three years. To take part in the program, students are obligated to attend school full time 
and to participate in essential program services. The original CUNY ASAP model included the 
following components. 


Student services: Students receive comprehensive advisement from an adviser with a 
small caseload, career information from a career and employment services staff member, and 
tutoring services separate from the usual college tutoring services. Each of these services is pro- 
vided by ASAP-dedicated staff members who work only with students in the program. ASAP 
advisers, who have caseloads of 60 to 80 students,* offer support related to a wide variety of 
academic and personal topics, including academic planning, balancing school with other respon- 
sibilities, accessing campus services, interacting with professors, staying on track to graduate, and 
dealing with personal issues. Students are required to meet with their adviser twice per month and 
to meet with career services staff members once per semester. Students in remedial courses or on 
academic probation must participate in tutoring once per week. 


Course enrollment: Students may enroll in blocked courses, or courses scheduled 
back to back, in their first year. This arrangement of classes has two goals. The first is mini- 
mizing the time required on campus so that students, many of whom work, can enroll full time. 
The second is ensuring ASAP students can take some of their classes with other ASAP students, 
fostering a sense of community. During the first few semesters, students also enroll in an ASAP 
seminar that covers topics such as goal-setting, study skills, and academic planning.° Students 
can also register for courses early so that they can create convenient schedules and get seats in 
the classes they need. 


3One notable exception is the recent study of traditional remedial math courses, compared with corequisite 
remediation, in which students take college-level math but are provided with additional academic supports. A 
randomized controlled trial at three CUNY colleges found that corequisite remediation led to an increase in 
course pass rates as well as an 8.1 percentage point increase in three-year graduation rates (Logue, Douglas, and 
Watanabe-Rose, 2019). 

4This section describes the model at the time of the evaluation; it has evolved since then as it has been 
expanded to serve more students. The advising requirements, for example, have been modified somewhat, and 
caseloads are currently 150 students per adviser. 

The current model delivers the content of the seminar in group advising sessions, rather than in a larger 
classroom setting. 


Financial supports: Students are given a tuition waiver that fills any gap between federal 
and state financial aid and college tuition and fees. Students also receive free use of textbooks and 
free MetroCards for use on public transportation. 


Requirements: Students are required to attend college full time during the fall and spring 
semesters and are encouraged to take courses during the winter and summer sessions. ASAP staff 
members also encourage students to complete developmental education early and to graduate 
within three years. 


Variations and adaptations: When the model was replicated in Ohio, some adjustments 
were made to fit the local context and to align with the evolved version of the ASAP model that 
CUNY was operating at the time. For example, CUNY offered program group students an ASAP- 
only student success seminar in their first few semesters. In contrast, the Ohio program did not 
create a special seminar course for program group students but enrolled these students into spe- 
cific sections of existing student success courses. The Ohio program also did not offer monthly 
metro cards to students (worth over $100 in New York), given the lower use of public transpor- 
tation in the state; instead it provided $50 gas/grocery gift cards to participating students. 


Another difference was that CUNY provided free textbooks to students in ASAP. AI- 
though one Ohio college offered free textbooks, two of the colleges instead provided students 
with vouchers of up to $300 per semester to cover the cost of textbooks. 


The Ohio program used a triage model of advising, in which the advising requirements 
differed depending on the students’ academic standing.* Students in the high-need group were 
required to meet with their assigned adviser twice per month, while other students could meet less 
often, although they usually had to see their adviser once per month. 


Tracking: Both the New York and Ohio programs used a management information sys- 
tem, created specifically for the program, to track student outcomes, participation, and progress. 
CUNY provided training to staff members at colleges in both sites on how to track and use these 
data for monitoring and improving program outcomes. 


Participating Colleges 


ASAP was tested at 3 of the 24 colleges in the CUNY system. Each college serves from 
15,000 to 20,000 students, making these the largest of CUNY’s community colleges. The three 
schools in this study were willing to participate in a randomized controlled trial, in which eligible 
students would be randomly assigned to either a program group that was eligible to participate in 
ASAP or a control group that was eligible for the college’s standard services. 


The ASAP Demonstration in Ohio was tested at 3 of the state’s 23 community colleges. 
One of the colleges serves about 20,000 students, while the other two are smaller, with 7,000 to 
8,000 students. These colleges were selected primarily based on administrators’ support for the 


°As noted earlier, one of the ways in which CUNY refined its model was to move to a triage model for 
advising, as implemented in Ohio. 


program, their willingness to take part in a random assignment evaluation, and their capacity to 
reach the desired sample size goals. See Appendix Table A.1 for selected characteristics of the 
six colleges. 


Implementation 


In order to ensure strong implementation and fidelity, the CUNY Office of Academic 
Affairs, which administered the program along with the participating colleges, monitored the op- 
eration of the program at CUNY. It also provided technical assistance to the Ohio colleges, lead- 
ing a workshop on the program model, helping the colleges develop action plans for implemen- 
tation, and providing regular and continuing direct technical assistance and training in the ways 
each program component could be enacted. 


Implementation research was conducted regularly throughout the three-year program pe- 
riod in each study. Overall, ASAP was well implemented in both New York and Ohio, and the 
program created a substantial service differential between students who were in ASAP and those 
who were eligible for the colleges’ usual services. 


Figure | presents students’ self-reported use of three key program services from a survey 
administered to all study participants in both states one year after program entry. Panel A presents 
the effects on incidence, or whether the student ever used a particular service during the first year. 
Panel B presents the effects on the intensity of use, or the number of times the student used the 
services, by semester. 


First, consider advising. The data in Panel A show that the program led to a moderate 
increase in the percentage of students who ever met with an adviser, in part because most students 
meet with an adviser at least once. The notable difference is in the intensity of advising, where 
program group students met with an adviser on average 15 times during the second semester, 
compared with 3 times for control group students. 


For tutoring and career services, the program led to larger increases in incidence, in part 
because fewer students use these services in the absence of the program. ASAP increased the 
number of students who ever used tutoring by 24 percentage points, for example, and program 
group students participated in nine sessions on average during the second semester, compared 
with three sessions for students in the control group. These averages are for the entire program 
and control groups, including zeros for those who never participated. Thus, the number of tutoring 
sessions among those who ever used tutoring is much larger than is shown in the figure. 


In terms of financial aid, most students in the program across both states reported receiv- 
ing assistance with the cost of textbooks, as well as the monthly incentive (gift cards in Ohio and 
transportation cards in New York). Few students in either state used the tuition waiver, as federal 
and state aid typically covered the cost of tuition (not shown). 


ASAP also promoted communication between students and staff members. Students in 
the program group were more likely than their control group counterparts to receive messages 


from advisers and other staff members on the importance of attending full time and graduating 
within three years (not shown). 


Finally, blocked scheduling was implemented only partially. In New York, students were 
offered blocked courses at two of the colleges and were encouraged to take at least one of the 
courses in the block if they could not take the full set. Transcript data show that although only 
about one-third of program group students enrolled in a full block of courses, a majority of stu- 
dents were enrolled in at least one course with four or more other ASAP students. Blocked sched- 
uling was not formally implemented in Ohio given the difficulty of identifying overlapping 
courses for large groups of students. As discussed below, eligibility for the study in Ohio was 
expanded to include students with up to 24 earned credits, meaning that some of them had already 
completed required courses. As an alternative to blocked scheduling, advisers informally tried to 
guide program group students into courses with each other. Transcript data indicate that about 
half of the program group students in Ohio enrolled in at least one course with at least four other 
program group students. 


The Evaluation 


Participating Students 


Students were eligible for the program and study if they met several criteria: their family 
income was below 200 percent of the federal poverty level or they were eligible for a Pell Grant; 
they were new to college or had earned fewer than 12 credits in CUNY or fewer than 24 credits 
in Ohio; they were willing to attend college full time; and they were in a major that could be 
completed within three years.’ The two studies differed with respect to developmental education 
requirements. For CUNY, the evaluation only included students who had one or two develop- 
mental education requirements, although a separate quasi-experimental evaluation conducted by 
CUNY staff members included students without these requirements (Strumbos and Kolenovic, 
2017). In Ohio, in contrast, eligibility was extended to students with or without developmental 
education requirements at study entry. Both programs excluded students with extensive (more 
than two) developmental education requirements, given that they were unlikely to earn a degree 
within three years. 


Students at CUNY were recruited for the study and randomly assigned just before the 
spring 2010 semester and the fall 2010 semester. Students in Ohio were recruited and randomly 
assigned just before the spring 2015, fall 2015, and spring 2016 semesters. The study samples 
include 896 students for CUNY and 1,501 students for Ohio.’ 


’The largest set of majors excluded was in allied health fields, such as physician’s assistant and nursing. 

8The total number of students randomly assigned was 903 students at CUNY and 1,522 students in Ohio. 
The analysis samples exclude students who withdrew from the study or whose consent form could not be recov- 
ered (7 students at CUNY and 21 students in Ohio). 


Table 1 outlines selected characteristics of the sample at baseline and, for comparison, 
characteristics of community college students nationwide.’ The students in the Ohio study dif- 
fered from those in New York in several ways. More Ohio sample students were nontraditional 
students—that is, they were older or had children (although they were less likely to work full 
time, another marker of nontraditional status). The New York sample also includes more Hispanic 
students and fewer White students than the Ohio sample, reflecting in part the differences in the 
demographic makeup of the participating cities. Fewer Ohio sample students had developmental 
requirements than did CUNY sample students, as expected given the different eligibility criteria 
for study entry in each state. 


The pooled sample is shown in the third column of Table 1. The sample is racially di- 
verse, with no racial majority. Just over half of the sample is Black or Hispanic. Fifty-one percent 
of the students were of “traditional” college age (aged 19 or younger), and just over 40 percent 
were nontraditional students. 


The students participating in the evaluation are roughly similar to community college 
students nationwide, with some exceptions. The majority of the pooled sample and the national 
sample are women, a sizable fraction have children, many are working, and the large majority 
have developmental education requirements. Students nationwide, however, are older than the 
students in the study sample and are more likely to work full time. The program’s eligibility 
guidelines may have led to these differences, given that the full-time attendance requirement may 
have discouraged full-time workers, who tend to be older students, from enrolling in the study. !° 


Data and Methods 


This paper uses data from several sources. First, students in the studies in both states filled 
out a baseline survey before random assignment, providing demographic and other information. 
Data on credit and degree completion at the participating colleges and at colleges within the 
broader CUNY and Ohio systems were obtained from transcript data provided by CUNY and the 
three participating Ohio colleges. Data on enrollment and degree receipt were also obtained from 
the National Student Clearinghouse, which covers students’ enrollments in nearly all postsecond- 
ary institutions throughout the United States, not just the participating colleges. 


Finally, surveys were administered to study participants in both sites approximately one 
year after they were randomly assigned. The survey, fielded to the full study samples in both 
states, covered topics such as sample members’ participation in and experiences with student 


*The national averages reflect students enrolled in public two-year institutions. 

‘Part of the difference in age and work status may be due to the fact that the national numbers include 
students seeking certificates and in nondegree programs, although they make up only about 15 percent of 
students. 


services, expectations of and engagement in college, employment, and financial aid and other 
financial issues. !! 


We use the following equation to estimate the effects of being offered ASAP: 


where Y;; denotes the outcome for Individual i in College j. 7; is an indicator variable for 
assignment to the program group, eligible for ASAP, and the vector Xj includes individual co- 
variates that are expected to be correlated with the outcomes of interest, such as race, age, gender, 
and developmental education requirements. The vector C includes randomization block indica- 
tors, one for each campus and cohort combination. The coefficient B is the key parameter of in- 
terest and represents the effect of the program offer. Since individuals were assigned at random 
to receive the offer to enroll in ASAP, this regression-based estimator of £ is unbiased with re- 
spect to our target of inference (the estimand), which is the average effect across individuals 
in the evaluation sample. '? 


Key outcomes include enrollment in two- and four-year colleges and degree receipt 
(associate’s degrees and bachelor’s degrees). The evaluation estimates the effects on these out- 
comes of the offer to participate in ASAP, or intent to treat estimates. Treatment-on-the-treated 
estimates are not presented, given that over 90 percent of students in the program group experi- 
enced at least one semester of the program. However, these estimates (of the effect of program 
participation, rather than of eligibility) can be viewed as roughly 10 percent larger than the esti- 
mates shown in this paper. Finally, no weighting is applied when estimating effects for the pooled 
sample. 


Earlier Findings 


ASAP’s effects in both states, through Year 3 in Ohio and Year 6 in New York, have 
been reported separately in earlier publications (Miller, Headlam, Manno, and Cullinan, 2019; 
Weiss, Ratledge, Sommo, and Gupta, 2019). The CUNY program was found to increase three- 
year degree receipt by 18 percentage points, from 22 percent for the control group to 40 percent 
for the program group. Rates of degree receipt continued to increase through Year 6 for both the 
program and control groups, and the differential between the two groups diminished in size. How- 
ever, the impact of the program at the end of Year 6 was still sizable, at 10 percentage points 
(p < 0.01). ASAP also increased enrollment rates and credit accumulation during the program 
period, or the first three years. 


‘Response rates for the surveys in New York and Ohio were 83 percent and 68 percent, respectively. Sep- 
arate analyses show that program and control group respondents were balanced on background characteristics 
(see Scrivener et al., 2015; Miller, Headlam, Manno, and Cullinan, 2019). 

'2Farlier reports documented that there were no systematic differences between the research groups for each 
study (Scrivener, Weis, and Sommo, 2012; Miller, Headlam, Manno, and Cullinan, 2019). Similarly, for the 
pooled sample, a regression model of program group status on a range of individual-level characteristics showed 
no significant differences between the research groups (not shown). 
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The Ohio program similarly led to large effects on enrollment, credit accumulation, and 
degree receipt by the end of Year 3. At that point, 35 percent of the students in the program group 
had earned a degree, compared with 19 percent of students in the control group, for an increase 
of 16 percentage points. 


Update on CUNY ASAP Effects Through Eight Years 


The earlier paper on CUNY ASAP showed that the effects on associate’s degree receipt dimin- 
ished over time, but remained sizable at the end of six years (Weiss, Ratledge, Sommo, and Gupta, 
2019). That paper was also the first to show positive effects on the receipt of bachelor’s degrees, 
with a positive effect in Semesters 8 and 9. However, this effect disappeared over time, as the 
control group caught up with the program group. This section updates those earlier findings to 
assess whether the stability in both levels and effects that was apparent by the end of Year 6 
continued through Year 8. 


Table 2 presents effects on enrollment and degree receipt at two- and four-year colleges 
(also referred to as associate’s degree-granting colleges and bachelor’s degree-granting col- 
leges). In Semesters 2 through 4, during the program period, enrollment rates in two-year colleges 
are higher for the program group than the control group, differences that are statistically signifi- 
cant. In the first two semesters of the post-program period (or Semesters 7 and 8), enrollment in 
two-year colleges is lower for the program group than the control group, given their higher rates 
of degree receipt and higher rates of enrollment in four-year colleges. However, by the last few 
semesters of the eight-year period, enrollment rates at two-year colleges have flattened out and 
fallen to less than 5 percent for both research groups. Enrollment in four-year colleges has also 
flattened out by that time, at around 15 percent, with no meaningful differences between the two 
groups during the last two years of follow-up. 


Effects on degree receipt are separated into associate’s and bachelor’s degrees. The main 
goal of the program is to increase the receipt of associate’s degrees. A few more students earned 
associate’s degrees over the additional two years of follow-up, but estimated effects on this out- 
come are stable in Semesters 12 through 16, at about 12 percentage points. 


In terms of bachelor’s degree receipt, about 25 percent of both research groups had earned 
a degree by the end of Year 8. The program led to small and statistically significant increases in 
Semesters 8 through 10, but had no effects thereafter. 


The findings suggest that the program’s primary effect was to speed up the receipt of 
bachelor’s degrees among students who would have earned them anyway. In contrast, the stability 
of the estimated impact on of associate’s degree receipt during the last six semesters of follow-up 
increases confidence in the idea that ASAP increased associate’s degree receipt among a sizable 
group (12 percentage points) of students who would not have earned an associate’s degree other- 
wise. The fact that the estimated effect on associate degree receipt peaked at 18 percentage points 
implies that, at a mintmum, ASAP’s effect of speeding up degree receipt, among those who would 
have graduated anyway, is at least 6 percentage points This speeding up of degree receipt is 
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notable and can reduce costs for colleges and students; however, the long-term increase in degree 
receipt is what can drive the benefits of the program, given that the gains from higher education 
accrue over an individual’s lifetime. 


Findings from the Pooled Sample 


The remainder of this paper examines effects through three years using the New York and Ohio 
samples, estimating effects for the pooled sample, for subgroups, by state and by college. This 
section describes effects for the combined sample. 


Figure 2 presents the results for enrollment and degree receipt at any two-year college.'° 
Enrollment rates at any school fall steadily after the first semester, but they fall less rapidly for 
students in the program group. By Semester 2, for example, only 73 percent of students in the 
control group were still enrolled, compared with 84 percent in the program group, a difference of 
11 percentage points. Effects are similar in size in Semesters 3 and 4 but become negligible after 
that point. 4 


Study participants begin earning degrees by Semester 3, and the increase is more rapid 
for students in the program group. By the end of Semester 6, degrees (nearly all associate’s de- 
grees) had been earned by about 36 percent of students in the program group, compared with 19 
percent of students in the control group—a substantial increase of almost 17 percentage points 
(with a 90 percent confidence interval ranging from 14 percentage points to 20 percentage points). 
As noted, this effect is nearly a doubling of degree receipt and is among the largest observed to 
date for a college intervention. The estimate of program effect is also for a broader population. 


Enrollment rates fall over time as students either graduate or leave school without a de- 
gree. Thus, the program’s effects on enrollment are somewhat difficult to interpret in later semes- 
ters, given that they are estimated for the full sample and include students who have graduated. 


Although not shown, enrollment rates in four-year colleges gradually increase each se- 
mester, and almost 14 percent of control group students are enrolled in Semester 6. The program 
increased that rate by an estimated 6.5 percentage points (p < 0.001). 


Overall, the rate of transfer to four-year colleges for both research groups is fairly low, 
although perhaps this is not surprising in light of the barriers to entry faced by low-income stu- 
dents. National data indicate that only about 33 percent of community college students transfer to 
a four-year institution within six years, despite the fact that the majority of them (as with this 
study sample) enter college intending to pursue a bachelor’s degree (Jenkins and Fink, 2016). 


Estimated effects are shown for the full study samples. Effects on these outcomes estimated for the survey 
respondent samples, used later in the paper, are very similar to those shown here. 

'4Prior evaluation reports also presented effects on full-time enrollment, given that it was a program require- 
ment. Impacts on full-time enrollment, not shown here, were large (15 to 20 percentage points) during the first 
four semesters and diminished thereafter. 
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Effects for Subgroups of Students 


As described above, for the pooled sample, the findings show that ASAP led to a notable increase 
in degree receipt as well as an increase in four-year college enrollment by the end of Year 3. This 
section examines whether effects vary for different types of students. Community college students 
are a diverse group, varying in their levels of academic preparation upon entry, their age, their 
family status, and their work status, for example. A large number of them are “nontraditional” 
students, meaning that they do not fit the profile of the single, childless, young adult entering 
college straight from high school. Subgroup analyses help to assess whether ASAP can be effec- 
tive for a wide variety of student types, which speaks to the generalizability of the findings. 


Table 3 displays the variations in effects on degrees earned and enrollment in four-year 
colleges, after three years. The top rows present effects by developmental education requirements 
at study entry. This subgroup was prespecified in each study as the sole confirmatory subgroup, 
as it is a central question of interest for which the study teams desired rigorous evidence. The 
remaining rows present effects for several subgroups as part of an exploratory analysis. Tests of 
effects across these dimensions (gender, age, high school diploma status, race/ethnicity, and work 
status) are of interest in their own right, but any findings should be interpreted with some caution 
and as suggestive for further study. 


The first set of columns shows effects on degree attainment, and the second set of col- 
umns shows effects on four-year college enrollment after three years. ASAP increased degree 
receipt for all subgroups examined, and the effects for each group are large and statistically sig- 
nificant. Although estimated effects are somewhat smaller for certain groups, such as those with- 
out a high school diploma at entry and Hispanic students, they are not statistically significantly 
different from estimated effects for their subgroup counterparts. In other words, there is no clear 
evidence that ASAP was more effective for one subgroup compared with another. 


Effects on four-year college enrollment are also positive for all groups, although they are 
not always statistically significant. There are two notable differences, although again, the varia- 
tion across groups is not statistically significant. First, there are smaller estimated effects for stu- 
dents without developmental education requirements, perhaps because these students were more 
likely to attend a four-year college in the absence of the program, as shown by rates for the control 
group, although the small sample size for this group makes the estimate imprecise. '* The program 
also did not appear to increase enrollment rates in four-year colleges for students without a high 
school diploma at entry. Again, this finding may be meaningful, or it may simply reflect an im- 
precise impact estimate for a very small subgroup. Similar to other students, the majority of stu- 
dents without a high school diploma reported at study entry that they expected to obtain at least a 
bachelor’s degree. Thus, the lack of effect for this group is not due to lower aspirations. 


'SThe subgroup consists entirely of students from Ohio because the New York evaluation focused only on 
students with developmental education requirements. 
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Effects by State 


The ASAP model was created by CUNY in 2007 and was operating at six of its community 
colleges before the evaluation began in 2009. The Ohio colleges worked closely with CUNY on 
the design and implementation of the program, but as expected, their replication was not exact. 
In addition, as shown below, the Ohio programs led to a smaller service contrast than the CUNY 
program did. These factors, along with the different contexts in which the program was tested, 
warrant a comparison of effects across the two states. 


Table 3 presents the academic outcomes, service contrasts, and student engagement rates 
for CUNY ASAP and the Ohio Demonstration programs. The first two rows show the effects on 
degree receipt and four-year enrollment, using data from administrative records for the full sam- 
ple. The next set of rows shows data from the survey subsample on the receipt of key program 
services (those shown for the pooled sample in Figure 1) and perceptions of support at school.'° 
The program had very similar estimated effects on both outcomes. CUNY ASAP increased de- 
gree receipt by 17.3 percentage points, compared with 15.9 percentage points for Ohio. Effects 
on four-year college enrollment are also very similar. 


As noted in the introduction, the similarity of effects is striking, given the difficulty of 
replicating original findings (the “replication crisis,” as it is sometimes called). But the similarity 
is also remarkable in light of the fact that the students, contexts, and service contrast differed 
across the two sites. For example, more students in Ohio than in New York were nontraditional 
students—that is, they were older or had children. This difference in the sample is notable, as 
there was some concern that the model, with its full-time attendance requirement, would not work 
for these types of students. Further, fewer students in Ohio than in New York had developmental 
education requirements. 


The context also differed in terms of management structure. CUNY is the largest urban 
community college system in the nation and is fairly centralized. Management of the program 
was led centrally by a dedicated ASAP team in the CUNY Office of Academic Affairs. Ohio’s 
community colleges, in contrast, are decentralized, meaning that the colleges largely implemented 
the program independently. 


Finally, the similar findings are also noteworthy given that the program in New York 
appears to have created a larger service contrast than the program in Ohio (as indicated by stu- 
dents’ responses to a survey). For example, program group students in New York met with their 
adviser on average 17.3 more times in the first semester than their control group counterparts did, 
compared with a difference of 8.3 visits in Ohio. Control group advising visits were fairly similar 
across the two states, suggesting that the difference in effects is not driven by different baseline 
levels. Ohio had implemented a triage model of advising, although this variation would primarily 
affect advising sessions in the second and later semesters. In addition, Ohio advisers had much 
larger caseloads than CUNY advisers did at the time of the study. 


‘CF ffects on academic outcomes were estimated for the survey respondent samples and are very similar to 
those shown in the table, which are based on the full samples. 
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Differences in impacts on tutoring are not as large, but they are still sizable. Part of this 
difference could be driven by the fact that the CUNY sample consisted entirely of students with 
developmental education requirements, for whom tutoring was required under ASAP, whereas 
only 75 percent of the Ohio students had developmental education requirements. However, when 
the analysis for the Ohio study was restricted to students with developmental education require- 
ments, the pattern of results was similar to that shown in Table 3.'” 


The final two rows of Table 4 depict the effects on students’ perceptions of support. 
ASAP in New York led to much larger changes in this area. That program, for example, led to a 
32.7 percentage point increase in the number of students who felt that they had adequate supports 
and services to succeed in school. In contrast, the increase in Ohio was 10.2 percentage points. 
Part of the difference in effects may be due to the relatively high level of support already felt by 
students in Ohio, as indicated by responses for the control group students, leaving less room for 
improvement. !* 


The findings suggest that there may be diminishing returns to increased participation in 
a given student service. In other words, perhaps encouraging students to visit their adviser an 
additional eight times is sufficient to generate the observed increase in graduation rates, with more 
visits contributing little added effect. The next section looks at this issue from another angle by 
assessing effects for the individual college campuses. 


Effects by College Campus 


Although the New York programs were managed by CUNY, the programs in both states were 
ultimately specific to a given campus, with its particular environment, program director, advisers, 
other staff members, and services. This section describes the effects by campus. The estimates 
for the pooled sample and by state provide an overall average estimate of ASAP’s effectiveness. 
Estimating effects by campus provide further evidence for the replicability of the model in differ- 
ent settings. A caveat to this analysis is the smaller sample sizes at each individual campus, lim- 
iting the statistical power of tests of individual site effects and tests for variation across sites. 


Effects on associate’s degree receipt through three years are shown in Figure 3. The esti- 
mates range from 5 percentage points to 21 percentage points, and six of the seven estimates are 
greater than 11 percentage points and statistically significant, as illustrated by the confidence in- 
tervals. Although there is some variation in estimated effects across the campuses, it is not more 
than would be expected by chance if the true effects were the same across the campuses (p = 
0.486 for the test of cross-campus differences in effects). The fact that the null hypothesis of 


"Impacts on advising and tutoring visits for that subsample, for example, were 7.6 visits and 6.0 visits, 
respectively. 

'8The differences in control group levels and effects between the two studies might also be due to differences 
in response rates to the survey across the two studies—83 percent for CUNY versus 68 percent for Ohio. How- 
ever, when the survey analyses were weighted to account for differential nonresponse, the findings were similar 
to those shown here. 
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similar effects cannot be rejected is encouraging and provides stronger evidence that the model 
can work in a range of settings. 


Figure 4 presents estimates by campus of the service contrast. Specifically, it shows esti- 
mated effects on the number of times students participated in the three primary services: advising, 
tutoring, and career services. It also depicts effects on the students’ average feelings of support at 
the campus (estimated as an average of the impact on the two student engagement outcomes 
shown in Table 3). The campuses are ordered in each graph by the size of their estimated effects 
on degree receipt, with the smallest effects on the left side and the largest effects on the right side. 


The figure shows notable variability in service contrast, which is statistically significant 
for each outcome. For example, the program-induced increase in advising was 20 visits at Cam- 
pus 7, compared with seven visits at Campus 2. The differences in impacts on feelings of support 
are especially large. 


The variation across campuses in service contrast raises the question of whether these 
effects are correlated with effects on degree receipt. Figure 5 displays the associations between 
degree attainment and the use of advising, tutoring, career advising, as well as between degree 
attainment and perceptions of overall support. Each panel in the figure plots estimated impacts 
on degree receipt (on the vertical axis) against a given estimated service contrast (for example, 
increase in the number of advising sessions attended) on the horizontal axis. The size of each 
bubble is proportional to the size of the sample on which it is estimated, with larger sites having 
larger dots. In addition, a weighted least squares regression line is estimated over the seven ob- 
servations and plotted in each graph, in which estimates based on larger samples are given greater 
weight. 


Note that this analysis is nonexperimental, and any association observed does not neces- 
sarily imply that the service contrast caused the impact. Other factors about the campuses, which 
may be associated with both outcomes, are not controlled for in the analysis. For example, the 
colleges vary in size, with some serving over 20,000 students and others fewer than 10,000 stu- 
dents. (Appendix Table | presents characteristics of the colleges, rather than the individual cam- 
puses, given that campus-level data are not available.) Other differences include the demographic 
composition of the student population in terms of race/ethnicity and age, and the rate at which 
students at a given college graduate. A range of campus-level features might contribute to pro- 
gram impacts, although the number of campuses in this study is too small to support a formal 
investigation. 


The panels highlight two points. First, the regression lines suggest that there is a positive, 
albeit weak, association between some estimates of service contrast and degree impacts. Effects 
on degree receipt tend to be larger, for example, the larger the effects on advising visits and career 
services visits. Similarly, effects on degree receipt are larger when there is a larger effect on stu- 
dents’ feelings of support. 


Second, however, despite the weak positive relationships, effects on degree receipt are 
still quite large even with substantially smaller service contrasts. For example, in the relationship 
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between ASAP’s effect on advising visits during the first semester and ASAP’s effects on degree 
completion, the weighted regression line suggests that the cluster of three campuses where the 
program caused an increase of five to eight additional advising visits had degree effects of around 
15 percentage points. The three campuses where ASAP caused an increase of over 16 additional 
advising visits had degree effects closer to 17 percentage points (this average includes the one 
outlier campus). Thus, from a cost perspective, there may be room to reduce the resources put 
into service contrast (such as the number of advising visits) and still achieve large effects. Put 
differently, the question might be: How does the cost of moving from 5 additional advising visits 
to 18 additional visits, for example, compare with the benefits of increasing degree receipt by an 
additional 3 percentage points? 


In sum, the findings by state presented in the previous section suggest that beyond some 
initial positive amount, additional impacts on students’ interactions with staff members may not 
be necessary to achieve large effects on degree receipt. However, the more disaggregated findings 
offered here support the idea that there is some positive relationship between service contrast and 
degree receipt. This analysis is only suggestive, however, given the small number of campuses, 
the small sample size for each campus, and the fact that the studies were not designed to estimate 
causal relationships between these two factors. A more formal analysis is warranted when data 
are available from subsequent studies. 


Conclusion 


This paper has presented an update of findings from CUNY ASAP through eight years and a 
synthesis of findings through three years from Ohio and New York. The longer-term findings for 
New York show that the effects on associate’s degree receipt narrow over time, as the control 
group catches up with the program group, but the effects nevertheless remain sizable and stable 
by the end of the eighth year, at 12 percentage points. Effects on four-year enrollment and bach- 
elor’s degree receipt narrowed completely by the end of the period, suggesting that the main effect 
of the program on these outcomes was to help a small subset of students achieve them more 
quickly. 


The pooled analysis for both states showed that through three years, the ASAP model 
had large positive effects on the receipt of associate’s degrees—36 percent of students who were 
offered the program graduated, compared with 19 percent of students who were offered typical 
college services. At the end of Year 3, the program had also increased enrollment in four-year 
colleges. The program increased degree receipt for a variety of student types, including women 
and men, those with developmental education requirements and those without, and those with a 
high school degree at study entry and those without a degree. The program also had large effects 
in a variety of settings. The consistency of these findings suggests that the results are likely to 
generalize beyond these students and colleges. 


An analysis by state also showed similar effects on degree attainment, even with some 
differences in implementation and service contrast. The Ohio programs led to smaller increases 
in advising and tutoring than CUNY ASAP did, for example, but they had similar-sized effects 
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on degree receipt. These findings raised the question of whether such large increases in student 
services, as observed for CUNY ASAP, are necessary to achieve substantial gains in graduation 
rates. However, the findings across the individual campuses suggest that there is some positive 
relationship between service contrast (or increases in advising and tutoring) and impacts on de- 
gree receipt. The issue of how program impacts vary with program components warrants further 
study, as it speaks to program improvement and cost. 


The findings reiterate that this model of comprehensive supports and requirements can 
lead to large and lasting increases in degree receipt. But providing these supports to students has 
a cost. The direct cost of the program in Ohio, for example, was about $1,800 per student per 
year. The cost of the original CUNY model was higher than that, at about $4,700 per student per 
year, in part because of higher New York salaries and expenses but also because of more intensive 
advising requirements and lower adviser caseloads. CUNY ASAP’s costs have gone down in 
recent years due to economies of scale as it has expanded to serve more students, and due to the 
adoption of a triage approach to advising and larger caseloads per adviser. In addition, although 
a formal analysis was not conducted for either study, some part of these costs would likely be 
recouped through formula funding in most states that increases with enrollment and degree re- 
ceipt. Of course, the costs of any intervention must be considered relative to any benefits it gen- 
erates. From a societal perspective, a primary benefit to consider is the increased earnings that 
come with an associate’s degree, which would accrue over the working lives of participants (Bel- 
field and Bailey, 2017). 


Despite the additional cost, CUNY has expanded ASAP substantially since the original 
evaluation. With funding from the city of New York, the model now operates at more colleges 
within the CUNY system and serves a broader group of students. In Ohio, in contrast, only one 
of the three colleges in the original study has sustained the program. Nationwide, colleges’ ability 
to implement and sustain ASAP will depend on funding support from the states or other sources. 


Another option for addressing the costs is to consider reducing the intensity of certain 
components or dropping certain components entirely. The study evaluated the effects of the com- 
plete package, making it impossible to identify which components were responsible for the ef- 
fects. Nonetheless, the Ohio results suggest that a triage approach to advising and tutoring— 
CUNY’s current approach—can work, and that the very large average increases in advising and 
tutoring usage observed in the CUNY evaluation may not be necessary to increase degree receipt. 
In addition, the Ohio colleges and one New York college did not formally implement blocked 
scheduling, raising the question of whether that component is necessary to the model. 


Other questions prompted by the findings are related to generalizability and scaling. One 
question is whether the model could be effective for a broader group of students. The eligibility 
requirements are fairly general, but the requirement to attend full time, for example, eliminated 
students who cannot attend at that level because of work, childcare, and other factors. (However, 
as noted, the ASAP model led to large increases in full-time enrollment, indicating that many 
students can attend full time with the necessary support.) The recently evaluated Stay the Course 
model did not require full-time enrollment and had large positive effects on degree receipt, 
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although the effects were primarily for women, and the small sample sizes mean that there is a 
fair amount of sampling error in the estimates. Nonetheless, another question is whether the full- 
time requirement could be modified in some way. Given the importance of full-time enrollment 
for making steady progress through school, the program would probably not want to give students 
who can attend full time the option of attending less (Adelman, 2004). But perhaps a version of 
the model could be tested that brings in and serves students who cannot manage the full-time 
schedule. '° 


Another group that might benefit from the program is students in fields of study that take 
longer than three years to complete, such as the large field of allied health. For good reason, given 
that the goal of ASAP was to increase three-year degree attainment, students in these types of 
fields were not a part of ASAP. However, it is likely that they would benefit from the compre- 
hensive supports and services provided by the ASAP model. Students with substantial remedial 
needs were also excluded from the program, since they too would be unlikely to graduate within 
three years. Perhaps some version of ASAP could help these students stay on track and earn a 
degree.”° 


Finally, although the findings shown here are very encouraging, the replication of the 
ASAP model in one other state is not definitive evidence of its broad effectiveness. More infor- 
mation is needed about the ability to implement the model and find similar effects in different 
places and for different types of students. Relatedly, CUNY provided critical technical assistance 
to the Ohio colleges and MDRC provided operational support, which likely helped with main- 
taining fidelity to the model. As CUNY continues to replicate the model and provide technical 
assistance, similar results may be expected.”! But as community colleges across the country de- 
velop their own programs borrowing from ASAP and without CUNY’s technical assistance, it 
will be important to understand if they can achieve similar results. 


'SCUNY is currently piloting an initiative at two of its community colleges for part-time students. The pro- 
gram incorporates elements of ASAP and other models. See http://www 1.cuny.edu/mu/forum/2018/06/26/get- 
ting-part-time-students-to-the-finish-line-bronx-and-laguardia-community-colleges-launch-programs-to-boost- 
graduation-rates. 

20CUNY Start, a program operating at eight of the system’s community colleges, is designed to help students 
with remedial needs prepare for college-level coursework. It is viewed as a pathway to the CUNY ASAP pro- 
gram. 

*!Replications of ASAP are being evaluated at Westchester Community College in New York, Blue Ridge 
Community College in North Carolina, and West Virginia University—Parkersburg in West Virginia. 
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Exhibits 


Table | 
Sample Characteristics for the CUNY ASAP, Ohio Programs, 


Pooled Samples, and National Population of Community College Students 


Characteristic 


Gender (%) 
Male 


Female 


Age (%) 
19 or younger 
20-23 
24 or older 


Average age (years) 


Race/ethnicity® (%) 
Hispanic 
White 
Black 
Other? 


Living with parents (%) 
Has children (%) 
First person in the family to attend college (%) 


Diplomas/degrees earned (%)° 
High school diploma 
High school equivalency 


Currently employed (%) 
Currently employed full time (%) 
Nontraditional student* (%) 


Has developmental education requirements (%) 


Sample size 


30 


CUNY ASAP Ohio Programs 


Sample 


38 
62 


22 
20 


Sample 


36 
64 


47 
22 
31 


23 


10 
46 
35 
10 


58 
27 


34 


87 
12 


60 
16 
47 


74 


1501 


Pooled 
Sample 


37 
63 


22 
27 


83 


2397 


(continued) 


National 
Sample 


44 
56 


32 


69 


68 


Table 1 (continued) 


SOURCES: Calculations using baseline information form data and placement test data from the CUNY 
Institutional Research Database (IRDB), and the ASAP Demonstration study colleges; IPEDS; and Stats in Brief, 
U.S. Department of Education, February 2017. 


NOTES: Italics indicate statistics calculated only for a subset of respondents. 

Distributions may not add to 100 percent because of rounding. 

’Respondents who said they were Hispanic and chose a race are included only in the "Hispanic" category. 

>The "Other" category includes Asian/Pacific Islander, Native American/Alaska Native, multiracial, and other 
races and ethnicities. 

‘Distributions may not add to 100 percent because categories are not mutually exclusive. A small percentage of 
sample members are excluded from these measures because they earned a certificate, technical or other degree and 
we are unable to determine if they had earned a high school diploma or equivalent. 

4Nontraditional students are defined as those who were 24 years old or older, worked 35 or more hours per 
week, had children, or did not receive a high school diploma and were not enrolled in high school at the time of 
random assignment. Students are listed as nontraditional if they fit any of these characteristics. 
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Table 2 


Associate's and Bachelor's Degree Enrollment and Receipt 


Two-Year Colleges Four-Year Colleges 
Control Std. Control Std. 
Characteristic Group Difference Error Group Difference Error 
Enrollment Rate (%) 
Semester 1 93.6 2.7 1.5 0.7 -0.2 0.5 
Semester 2 81.1 10.1 2.3 0.9 0.1 0.6 
Semester 3 68.9 6.9 3.0 1.5 0.0 0.8 
Semester 4 58.2 8.0 3.3 5.3 -0.1 1.5 
Semester 5 43.8 1.5 3.3 12.6 4.0 2.4 
Semester 6 29.9 -3.1 3.0 17.8 22. (2.7 
Semester 7 19.8 -7.6 2.5 23.0 46 2.9 
Semester 8 15.3 -6.6 2.2 26.2 2.5 3.0 
Semester 9 11.5 -3.6 2.0 26.9 -0.4 3.0 
Semester 10 8.0 -2.0 1.7 27.2 -0.1 3.0 
Semester 11 6.0 0.9 1.6 26.2 54 2.8 
Semester 12 4.9 0.2 1.5 21.9 -2.9 2.7 
Semester 13 4.6 -0.6 1.4 19.9 -1.7 2.6 
Semester 14 5.1 -1.6 1.3 18.1 -2.1 2.6 
Semester 15 4.5 -0.3 1.4 12.9 2.3 2.3 
Semester 16 25 0.6 1.1 13.0 2.4 2.4 
Graduation Rate (%) 
Semester | 0.0 0.0 0.0 0.0 0.0 0.0 
Semester 2 0.0 0.2 0.2 0.0 0.2 0.2 
Semester 3 1.1 1.8 1.0 0.0 0.2 0.2 
Semester 4 8.9 5.6 2.2 0.0 0.2 0.2 
Semester 5 15.7 13.4 2.7 0.0 0.5 0.3 
Semester 6 22.1 17.9 3.0 0.0 0.9 0.5 
Semester 7 26.5 15.8 3.1 0.8 0.7 0.7 
Semester 8 29.7 15.2 3.2 2.2 2.2 1.2 
Semester 9 31.9 144 3.2 4.0 3.1 1.5 
Semester 10 35.1 12.0 3.3 8.3 3.6 2.0 
Semester 11 36.1 12.4 3.3 12.9 2.0 2.3 
Semester 12 37.5 12.2 3.3 16.7 0.8 2.5 
Semester 13 38.3 11.9 3.3 18.0 14 2.6 
Semester 14 38.8 12.3 3.3 20.6 16 2.7 
Semester 15 39.2 12.5 3.3 22.4 0.7) 2.8 
Semester 16 39.9 12.1 3.3 23.5 14 2.9 
Sample size 445 896 445 896 


~ SOURCES: Calculations using data from the CUNY Institutional Research 
Database (IRDB), and the National Student Clearinghouse (NSC). 


NOTE: Estimates are adjusted by random assignment blocks and selected 
baseline characteristics. 
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Table 3 
Degrees Earned and Four-Year College Enrollments After Three Years: 
Variation in Effects by Student Characteristics 


Earned Degree Enrolled in a Four-Year College 

P-Value for P-Value for 

Control Std. Subgroup Control Std. Subgroup 

Characteristic Group Difference Error Difference Group Difference Error Difference 

Developmental education 

requirements (%) 0.189 0.445 
None 25.5 17.6 5.2 21.0 10 4.6 
One 22:5 21.3 3.3 14.8 7.6 2.8 
Two or more 16.2 13.5 2.6 11.8 6.9 2.2 

Gender (%) 0.206 0.739 
Female 20.9 17.9 2.3 15.1 6.0 2.0 
Male 19.2 13.1 2.9 12.1 7.0 2.5 

Age (%) 0.241 0.193 
19 or younger 20.0 17.6 2.5 14.9 7.6 2.2 
20-23 20.9 10.4 3.9 13.6 1.7 3.2 
24 or older 19.8 18.4 3.7 11.7 9.3 3.0 

High school diploma at study entry (%) 0.372 0.307 
No 17.5 13.0 4.2 13.0 3.0 3.6 
Yes 20.6 17.2 2.0 14.1 7.10 1.7 

Race/ethnicity (%) 0.305 0.754 
Hispanic 22.1 10.3 4.0 14.4 4.6 3.3 
White 23.7 19.5 3.4 13.3 74 2.7 
Black 14.9 18.4 3.0 12.9 9.0 2.7 
Other 18.1 18.5 5.9 15.3 5.3 5.2 

Employed at study entry (%) 0.736 0.939 
No 21.3 15.9 2.6 13.7 6.3 2.2 
Yes 18.7 17.1 2.6 14.1 6.5 2.3 


SOURCES: Calculations using data from the CUNY Institutional Research Database (IRDB), the 
ASAP Ohio Demonstration study colleges, and the National Student Clearinghouse (NSC). 


NOTES: Estimates are adjusted by random assignment blocks and selected baseline characteristics. 
Rounding may cause slight discrepancies in calculating differences and sums. 
The H-statistic test was used to test for statistically significant differences in impact estimates across 
different subgroups. 
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Table 4 


Academic, Service, and Engagement Effects, by State 


Outcomes 


Academic outcomes (%) 
Earned a degree 
Enrolled in a four-year college 


Service contrast 
Times met with adviser 
Times met with tutor 


Times met with career services 


Student engagement (%) 
Had an employee to turn to for advice 
Had supports/services needed 


Sample size 


22.7 
18.0 


3.8 
4.2 
1.1 


58.4 
56.9 


445 


CUNY ASAP 
Control 
Group Difference Error 


17.3 
6.8 


17.3 
8.5 
3.8 


28.0 
32.7 


896 


Ohio Programs 


Std. Control 
Group Difference 


3.0 18.4 
2.7 11.8 
1.4 5.0 
1.1 3.4 
0.4 0.7 
3.2 72.6 
3.1 81.0 

695 


15.9 
5.7 


8.3 
4.9 
1.7 


16.5 
10.2 


1,501 


Std. 


Error 


2.3 
1.9 


0.9 
0.8 
0.3 


2.5 
2.2 


P-Value 
for Subgroup 
Difference 


0.725 
0.739 


0.000 
0.010 
0.000 


0.005 
0.000 


SOURCES: Calculations using data from the CUNY Institutional Research Database (IRDB), the ASAP 
Ohio Demonstration study colleges, the National Student Clearinghouse (NSC). and the MDRC student 


surveys. 


NOTES: Estimates are adjusted by random assignment blocks and selected baseline characteristics. 
Sample sizes for specific outcomes may vary because of missing values. 
Rounding may cause slight discrepancies in calculating differences and sums. 
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Appendix Table A.1 


College Characteristics for the CUNY ASAP and Ohio Programs Evaluation 


Cincinnati State 
Technical and 
Community College 


Characteristic 


Campus structure 


1 main campus + 3 other 
locations around the city 


Number of students 8,807 

Full-time students (%) 26 
Male (%) 44 
Black (%) 24 
Other minorities (%) 17 
Under 24 years of age (%) 55 

Campus setting City: large 
Geographic location Southwest Ohio 
Graduation rate (%) 15 

Undergraduate students receiving Pell Grants (%) 33 
Sample size (total = 2,397) 467 


Lorain County 
Community College 


1 main campus + 5 
satellite centers across 
the county 


11,042 
27 


39 


18 


70 


City: small 


Northeast Ohio 


23 


36 


513 


Cuyahoga 


Community College 


4 main campuses + 8 


satellite locations 


across the county 


23,900 

31 

40 

25 
20 

61 

City: large 
Northeast Ohio 
15 

38 


521 


Borough of Kingsborough 
Manhattan Community College 
Community College 


1 main campus 1 main campus 


22,534 18,606 
65 60 
41 44 
30 32 
58 33 
68 74 
City: large City: large 
New York City New York City 
14 25 
79 69 
401 325 


SOURCES: Study colleges' websites and MDRC calculations using data from the Integrated Postsecondary Education Data System (IPEDS). 
NOTES: Two campuses in this study, Tri-C East and Tri-C, West are combined into one college, Cuyahoga Community College. 
IPEDS data for campuses in each state reflect a year within each evaluation period: 2017 in Ohio and 2010 in New York. 
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LaGuardia 
Community College 


1 main campus 


17,569 
58 
41 
17 
69 
64 
City: large 
New York City 
15 
12. 


170 


100.00 11.0 


80.00 413 
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» 00.00 
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Advising Tutoring Counseling 
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Figure 1A. Ever Received Key Service 


Number of activities 


Semester 1 Semester 2 | Semester 1 Semester 2 | Semester 1 Semester 2 


Advising Tutoring Career services 


@Program group OControl group 


Figure 1B. Number of Times Students Received Each Key Service 


SOURCE: Calculations using data from the student survey. 


NOTES: Estimates are adjusted by random assignment blocks and selected baseline 
characteristics. 
Sample sizes for specific outcomes may vary because of missing values. 
Rounding may cause slight discrepancies in calculating differences. 
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—— Program group's enrollment rate in associate's degree-granting colleges 
— Control group's enrollment rate in associate's degree-granting colleges 
Program group's graduation rate from associate's degree-granting colleges 


Control group's graduation rate from associate's degree-granting colleges 


Figure 2. Enrollment in Associate's Degree-Granting Colleges and Associate's Degree Receipt 


SOURCE: MDRC calculations using data from the CUNY Institutional Research Database (IRDB) and the 
ASAP Ohio Demonstration study colleges. 


NOTE: The first number next to each pair of lines is the estimated effect in Semester 6. The second number, in 
parentheses, is the standard error. 
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Estimated effect (percentage points) 


Campus 


Figure 3. Associate's Degree Effects by College/Campus, Through Three Years. 


SOURCES: MDRC calculations using data from the CUNY Institutional Research Database (IRDB) 
and the ASAP Ohio Demonstration study colleges. 


NOTES: The figure shows estimated effects on associate's degree receipt by college, indicated by the 
squares and triangles, and 90 percent confidence intervals, shown by the vertical lines. The CUNY 
campuses are denoted by squares; the Ohio campuses are denoted by triangles. 

Estimates are adjusted by random assignment blocks and selected baseline characteristics. 
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Figure 4. Effects on Service Contrast 


SOURCES: MDRC calculations using results from the MDRC student surveys. 


NOTE: The figure shows estimated effects by college, indicated by the triangles and squares, and 90 percent 
confidence intervals, shown by the vertical lines. The outcomes in this figure are the number of times students 
engaged in a certain program activity in the first semester, except for the "feelings of support" outcome. The 
CUNY campuses are denoted by squares; the Ohio campuses are denoted by triangles. 

Estimates are adjusted by random assignment blocks and selected baseline characteristics. 
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Figure 5. Association Between Service Contrast and Impacts on Degree Receipt 
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